| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-12-03 | |||
| 18:07:07 | sean-k-mooney | does createimage upload as a snapshot i though it flattened the image | |
| 18:07:23 | sean-k-mooney | ok | |
| 18:07:56 | efried | rebuild is your same instance. Otherwise you're creating a new instance. If you're creating a new instance, it will get a new (or no) vTPM. | |
| 18:08:41 | efried | so the only quirk is that in order for rebuild to restore your vTPM, you have to use an image from createBackup. | |
| 18:08:43 | efried | which makes sense | |
| 18:08:59 | efried | if you use a random image from createImage, why would you expect your vTPM from your original instance to be restored? | |
| 18:09:23 | efried | conversely, if you're creating a brand new instance (from *any* image), why would you expect anything other than a fresh vTPM? | |
| 18:09:56 | sean-k-mooney | ya so i said previously that the vtpm should be tied to the lifetime of the instace so i think that makes sense | |
| 18:10:19 | efried | we've got an answer for all the move operations and shelve/unshelve. | |
| 18:10:35 | sean-k-mooney | we are explcitly saying the usecase of i create a vm snapshot and spawn 100 more is expresly out of scope (in that they wont all get the orginal vms vtpm) | |
| 18:11:08 | efried | I wouldn't phrase it as "out of scope". It's got clean, predictable behavior: you don't get the original VM's vTPM. | |
| 18:11:29 | sean-k-mooney | yes that is what i intened | |
| 18:11:37 | efried | which, for anyone using a vTPM, ought to make sense. "don't give my secure stuff to another instance" | |
| 18:11:48 | sean-k-mooney | copying the vtpm in that case is not supported intentionally | |
| 18:12:19 | efried | yeah, I'm just getting nitpicky about the language. "out of scope" and "not supported" imply "don't work". | |
| 18:12:28 | sean-k-mooney | so i would be ok with that set of constratits but im not really the person you have to convice | |
| 18:12:55 | eandersson | dansmith so we jumped from mitaka to rocky | |
| 18:13:09 | eandersson | So it's difficult to say exactly what part improved performance the most | |
| 18:13:17 | eandersson | but we believe placement played a big role here | |
| 18:13:35 | dansmith | eandersson: well, is it scheduling performance, instance listing performance, build time, etc? | |
| 18:13:40 | eandersson | scheduled is a lot better at least ~50% | |
| 18:13:49 | dansmith | sweet, so that's likely placement yeah | |
| 18:13:54 | eandersson | Also scaling computes | |
| 18:14:05 | dansmith | eandersson: what does "scaling computes" mean? | |
| 18:14:06 | eandersson | We initially limited ourself to 1000 computes per region | |
| 18:14:12 | eandersson | in Mitaka | |
| 18:14:23 | eandersson | but now believe we can reach a much higher number than that | |
| 18:14:55 | dansmith | eandersson: is that via cells or just in general? because I don't think we've done anything other than get *more* chatty to the db/mq since mitaka :) | |
| 18:14:57 | eandersson | And even if we do hit limitations we can now use cells | |
| 18:15:23 | eandersson | scheduler used to be really heavy on rmq | |
| 18:15:29 | eandersson | in mitaka | |
| 18:15:53 | sean-k-mooney | eandersson: do you run sepperate rmq instacne per openstack service | |
| 18:16:01 | eandersson | We do not | |
| 18:16:24 | eandersson | but even at 1k computes we are barely putting any stress on rmq at the moment | |
| 18:16:32 | eandersson | Of course we are not using ceilometer | |
| 18:16:34 | sean-k-mooney | im surpirsed you are getting to 1000 nodes on one cluster with neutron and nova sharing it | |
| 18:16:48 | dansmith | eandersson: so you're thinking that lowered load on rabbit from the scheduler lets you have more computes? | |
| 18:16:52 | sean-k-mooney | ya not using ceilometer helps | |
| 18:17:02 | eandersson | dansmith I think it helps | |
| 18:17:09 | dansmith | ack | |
| 18:18:12 | eandersson | We were getting a lot of slow api calls in mitaka and they are all very consistent now | |
| 18:19:06 | eandersson | The only problems we have with nova now is getting our super custom scheduling logic to scale | |
| 18:19:07 | dansmith | that's what I'm interested in specifically, | |
| 18:19:17 | dansmith | but without knowing which calls those were I can't really attribute them to anything | |
| 18:19:31 | dansmith | (that == slow api calls) | |
| 18:20:55 | eandersson | Yea - unfortunately a lot of the research and testing we did was back in ~2016 | |
| 18:21:48 | eandersson | We didn't do a great job tracking individual improvements when going from Mitaka to Rocky | |
| 18:21:53 | sean-k-mooney | eandersson: do you have a list of constraits you need to schduler on that are not supported cleanly upstream that could be shared perhaps we could accomadate some of your custom logic | |
| 18:22:44 | eandersson | We do flavor stacking and what... we internally call "perfect fit". | |
| 18:22:59 | sean-k-mooney | using the type affingity filter | |
| 18:23:07 | mriedem | you also have a variant of the old flavor affinity filter yeah? | |
| 18:23:08 | sean-k-mooney | so that each host only one flavor | |
| 18:23:30 | eandersson | So we actually want to stack flavors on a compute | |
| 18:23:34 | eandersson | because these are game servers | |
| 18:23:40 | eandersson | So each game server takes up one numa | |
| 18:24:07 | eandersson | but we still want to be able to schedule other micro services on top | |
| 18:24:50 | eandersson | Since we don't want to have to divide the fleet | |
| 18:24:55 | sean-k-mooney | so you want to pack the large flavor and then fit the micof servces wehre tehy can | |
| 18:24:59 | eandersson | Yep | |
| 18:25:17 | sean-k-mooney | that not really unresobaly to be fair | |
| 18:25:41 | sean-k-mooney | the main issue i guess you face right now is fragmenation | |
| 18:26:02 | eandersson | Yea - if we go with the out of the box implementation | |
| 18:26:04 | sean-k-mooney | e.g. a small instnace spawns preventing a large instnace | |
| 18:26:09 | eandersson | yep | |
| 18:26:13 | dansmith | but also...scheduler filters are the one place I think we *should* be pluggable, and so unless other people want *exactly* the same weird scheduling thing, it makes sense for them to do this on their own, IMHO | |
| 18:26:37 | eandersson | Yep - I hate that I have to patch nova for this | |
| 18:26:55 | eandersson | I mean we have a super custom way of deploying, plus probably 20 custom nova patches at least | |
| 18:27:02 | sean-k-mooney | this is not the first time i have heard this requrest however | |
| 18:27:17 | eandersson | So it's not a big deal, but would be a lot easier for us to manage it if it was pluggable | |
| 18:27:27 | dansmith | scheduler filters *are* pluggable | |
| 18:27:35 | sean-k-mooney | eandersson: as are the weighers | |
| 18:27:36 | dansmith | presumably you're dependent on other changes/ | |
| 18:27:46 | eandersson | They are? | |
| 18:27:50 | sean-k-mooney | yep | |
| 18:28:00 | sean-k-mooney | but its non ovious how to do it | |
| 18:28:09 | dansmith | it's very obvious | |
| 18:28:20 | dansmith | it may not be _documented_ :) | |
| 18:28:34 | eandersson | btw we also have weights for upgrading computes (e.g. a compute that needs a OS upgrade would be moved into an aggregate to reduce the changes of it getting scheduled to) | |
| 18:28:48 | dansmith | https://docs.openstack.org/nova/latest/user/filter-scheduler.html#writing-your-own-filter | |
| 18:29:00 | sean-k-mooney | eandersson: here is an example https://opendev.org/x/nfv-filters | |
| 18:30:15 | eandersson | Ah yea I see | |
| 18:31:08 | sean-k-mooney | you just do filter_scheduler.available_filters=nova.scheduler.filters.all_filters,nfv_filters.nova.scheduler.filters.aggregate_instance_type_filter | |
| 18:31:15 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Kill it https://review.opendev.org/696518 | |
| 18:31:15 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Rename 'nova.network.neutronv2' -> 'nova.network' https://review.opendev.org/696745 | |
| 18:31:16 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Rename 'nova.network.security_group.neutron_driver' -> 'nova.network.security_group' https://review.opendev.org/696746 | |
| 18:31:17 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove unnecessary 'neutronv2' prefixes https://review.opendev.org/696776 | |
| 18:31:17 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove unused exceptions https://review.opendev.org/697149 | |
| 18:31:17 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove db methods for ProviderMethod https://review.opendev.org/697150 | |
| 18:31:18 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove unused 'stub_out_db_network_api' https://review.opendev.org/697151 | |
| 18:31:18 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove remaining nova-network quotas https://review.opendev.org/697152 | |
| 18:31:19 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove use of legacy 'FloatingIP' object https://review.opendev.org/697153 | |
| 18:31:19 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove use of legacy 'Network' object https://review.opendev.org/697154 | |
| 18:31:20 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove use of legacy 'SecurityGroup' object https://review.opendev.org/697155 | |
| 18:31:20 | openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove unused nova-network objects https://review.opendev.org/697156 | |
| 18:32:07 | sean-k-mooney | you can similary have out of tree weighers that you can load in a similar way | |
| 18:34:19 | sean-k-mooney | dansmith: by the way this is actully a lot simpler then i rememeberd i was thinking of out of tree virt drivers which require you to reopen the nova namespace and do other things to make them work | |
| 18:34:39 | sean-k-mooney | dansmith: be we dont really want that to be plugablle in the same way so it makes sense that is more work | |
| 18:34:54 | dansmith | yup | |
| 18:49:33 | artom | Do we not reset the old flavor back on the instance if we fail a resize? Fail as in, something goes wrong during _prep_resize or resize_instance | |
| 18:49:51 | efried | gosh, I would hope we do | |