Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-11-30
16:13:45 openstack bug 1764883 in OpenStack Compute (nova) "Evacuation fails if the source host returns while the migration is still in progress" [Undecided,Fix released] https://launchpad.net/bugs/1764883 - Assigned to Lee Yarwood (lyarwood)
16:13:45 openstackgerrit Lee Yarwood proposed openstack/nova stable/pike: Add regression test for bug #1764883 https://review.openstack.org/621204
16:14:31 lyarwood temka: ^ there you go
16:16:32 Sundar efried: Please ping me on cyborg channel when you can. Thanks.
16:16:37 mnaser mriedem: sorry, i meant i have 0 disk flavors and i am ok with non-admins no longer being able to boot
16:17:38 temka lyarwood, ack
16:19:02 mriedem mnaser: yeah in my mind the upgrade check wouldn't warn on that - if the policy is the default, you're ok even with 0-disk flavors. i'd think the check would warn if you have 0-disk flavors and your policy was not the default (admin_api)
16:19:13 mriedem but like i said, the policy rule could be custom so it's hard to say
16:19:34 mnaser ah i see what you mean, i could have renamed by admin_api to foo_bar and it would warn anyways
16:19:41 mriedem right
16:19:51 mriedem that's why i'm hesitant to add something
16:20:10 mnaser + if someone is actually running their own policy
16:20:13 mnaser its very likely that its custom anyways
16:22:43 mriedem SteelyDan: before i make this change that leakypipes suggested in my cross-cell resize series, do you agree with it or a meh? https://review.openstack.org/#/c/614012/3/nova/db/sqlalchemy/models.py@794
16:24:58 SteelyDan mriedem: HMM
16:25:00 SteelyDan so,
16:25:17 SteelyDan on the one hand, expanding the schema for a boolean seems kinda silly when we could store something useful
16:25:48 SteelyDan BUT, the migration lives in the cell db, and I don't like ever telling the cell what cell it is, which it can kinda infer from that data
16:26:04 SteelyDan like, imagine you do some migrations and then later collapse or split a cell
16:26:49 mriedem as noted in the change, i get the source cell mapping from the host mapping tied to the source_compute in the migration object https://review.openstack.org/#/c/603930/15/nova/conductor/tasks/cross_cell_migrate.py@871
16:26:56 mriedem so i don't *need* the explicit uuid in the migrations table
16:27:00 SteelyDan right
16:27:07 mriedem the boolean flag is really for flow control
16:27:11 SteelyDan I know
16:27:37 mriedem i don't know if the source/dest cell uuids being in the migration record could be abused later
16:27:50 mriedem i mean, i'm sure they could, i'm just not sure in what case
16:28:21 SteelyDan well, it's not a matter of abuse so much as just separation of responsibilities
16:28:23 mriedem once the migration is done, we shouldn't really care about the record anymore, except reporting
16:28:45 mriedem and if an operator is going to split cells, they had better make sure they don't have migrations going on while doing so
16:29:06 SteelyDan yeah, not a matter of ongoing ones, but just those being wrong later
16:29:24 SteelyDan nothing else in the cell db knows about its position in the hierarchy and I think we should keep it that way
16:29:31 SteelyDan so while I was going to argue against just storing a boolean,
16:29:36 mriedem right. i mean, there could be a comment in there saying, "don't trust these after the migration is done"... but that's a bit hokey
16:29:39 SteelyDan I think I'd prefer that over the uuids just for that reason
16:29:43 SteelyDan heh
16:29:53 mriedem SteelyDan: ok care to reply so i can move on?
16:30:03 SteelyDan yep
16:30:08 mriedem thanks much
16:30:10 mriedem <3
16:38:42 openstackgerrit Corey Bryant proposed openstack/nova stable/pike: Revert "Backport tox.ini to switch to stestr" https://review.openstack.org/621227
17:09:01 openstackgerrit Matt Riedemann proposed openstack/nova master: Add Destination.allow_cross_cell_move field https://review.openstack.org/614035
17:09:01 openstackgerrit Matt Riedemann proposed openstack/nova master: Add Migration.cross_cell_move and get_by_uuid https://review.openstack.org/614012
17:09:02 openstackgerrit Matt Riedemann proposed openstack/nova master: Change HostManager to allow scheduling to other cells https://review.openstack.org/614037
17:09:02 openstackgerrit Matt Riedemann proposed openstack/nova master: Add InstanceAction/Event create() method https://review.openstack.org/614036
17:09:03 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Cross-cell resize https://review.openstack.org/603930
17:09:03 openstackgerrit Matt Riedemann proposed openstack/nova master: Add CrossCellWeigher https://review.openstack.org/614353
17:13:35 openstackgerrit Merged openstack/python-novaclient master: Add Python 3.6 classifier to setup.cfg https://review.openstack.org/621093
17:26:41 openstackgerrit Eric Fried proposed openstack/nova master: Reduce calls to placement from _ensure https://review.openstack.org/615677
17:26:42 openstackgerrit Eric Fried proposed openstack/nova master: Commonize _update code path https://review.openstack.org/615705
17:26:42 openstackgerrit Eric Fried proposed openstack/nova master: Consolidate inventory refresh https://review.openstack.org/615695
17:26:43 openstackgerrit Eric Fried proposed openstack/nova master: Rip out the SchedulerClient https://review.openstack.org/617049
17:26:43 openstackgerrit Eric Fried proposed openstack/nova master: Rip the report client out of SchedulerClient https://review.openstack.org/617042
17:26:44 openstackgerrit Eric Fried proposed openstack/nova master: Use a static resource tracker in compute manager https://review.openstack.org/620711
17:26:46 efried leakypipes: This all ought to be ready now ^
17:26:58 leakypipes efried: k
17:28:39 openstackgerrit Eric Fried proposed openstack/nova master: Turn off rp association refresh in nova-next https://review.openstack.org/616033
19:35:18 pacharya Hi I ran into weird issue where nova compute received empty port list for an instance during heal instance info cache. This caused the instance info cache for that instance to be set to empty list and now the APIs are not returning any network information. In the subsequent run neutron returned correct port list but nova does not fix the instance info cache so it remains broken. Any ideas on how to fix this?
19:36:06 SteelyDan mriedem: ^
19:40:21 mriedem pacharya: https://review.openstack.org/#/c/591607/
19:41:19 mriedem without ^ the heal_instance_info_cache and network-changed events just refresh the cache from the cache...which is why it doesn't help you
19:41:42 mriedem unfortunately you need to rebuild the cache somehow, and off the top of my head i think that might mean attaching/detaching a port to/from the server,
19:41:43 mriedem if possible,
19:41:55 mriedem could you just attach a dummy port to the server and then immediately detach it?
19:42:11 mriedem otherwise you might have to try migrating the server, but i'm not sure if that will rebuild the network info cache for the instance
19:44:07 mriedem looking at the code, attaching a new port might just put that port into the cache...
19:44:32 mriedem so i'm not really sure what to tell you beside look at https://review.openstack.org/#/c/591607/
19:46:13 mriedem i know we have https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-reset-network but i don't think that will help
19:58:03 pacharya Thanks ! @mriedem. I did try detaching and attaching the port but that did not seem to help. Will try it again just to be sure.
20:01:08 mriedem pacharya: as i said, i think that at most will just refresh the cache for that single port - but if that's the only port attached to the server then that should be enough
20:06:55 pacharya Yeah and you are also correct about reset-network. that does not work either. I will look into the fix that you pointed out
20:20:25 zzzeek cdent: so with https://review.openstack.org/#/c/614350/18, you tried putting the oslo fixtures into placement/tests/fixtures.py and it blew up ?
20:34:43 pacharya mriedem: will this get backported to queens https://review.openstack.org/#/c/591607/ ?
20:34:55 pacharya just curious
20:35:25 openstackgerrit Matt Riedemann proposed openstack/nova master: Use new ``initial_xxx_allocation_ratio`` CONF https://review.openstack.org/602804
20:35:26 openstackgerrit Matt Riedemann proposed openstack/nova master: Add compute_node ratio online data migration script https://review.openstack.org/609995
20:35:26 openstackgerrit Matt Riedemann proposed openstack/nova master: Add ratio online data migration when load compute node https://review.openstack.org/613499
20:36:00 zzzeek cdent: ummmm....it doesnt' seem to be provisioning anonymously named databases....odd
20:36:06 zzzeek wonder how it works
20:36:06 mriedem leakypipes: i have gone through the first 3 of 5 changes in that series, you might want to start warming up to it sooner than later since it's in a runway slot (for the 2nd time) and you are familiar given it was your spec
20:36:40 mriedem pacharya: idk, i haven't looked at it in awhile - maciej has taken it over
20:36:59 mriedem pacharya: i would think so though if we can
20:40:33 mriedem jackding: oh boy the io semaphore unit test changes are causing a race http://logs.openstack.org/22/606122/7/check/openstack-tox-py27/d70a4d5/testr_results.html.gz
20:48:10 mriedem looks like mdbooths new evacuate functional tests are also racy
20:56:11 cdent zzzeek: what's there is the result of much experimentation and it does work: i can witness the databases being created and manipulated (turned on query logging in the database logs)
20:56:24 cdent the names of things are being overridden
20:56:38 zzzeek cdent: OK so it makes anonymously named databases in PG / MySQL for the migration tests ?
20:57:32 cdent zzzeek: no, it skips that part because it was getting in way of managing multiople databases in the same process. also, as far as I could tell the anonymous naming was only working for mysql, not for postgresql
20:58:01 zzzeek cdent: do you run multiple tests concurrently in a single process?
20:58:50 sean-k-mooney zzzeek: in the nova tests we have eventlet so maybe
20:58:54 cdent on the postgres side is was always the same name, so given that worked for postgres, I made it work for mysql and sqlite too, and then locked the tests with an external lock so there aren't concurrency issues between processes. There aren't concurrent tests in the same process but there is concurrency inter-process
20:59:07 zzzeek cdent: yes that is all normal
20:59:13 cdent sean-k-mooney: we're talking about the placement side right now
20:59:20 zzzeek cdent: OK so the oslo_db way here is good to go, so I will see if i can make it work
20:59:37 zzzeek cdent: also there's just one enginefacade in placement, it seems
20:59:52 zzzeek cdent: which also makes this straightforwad, I see normal create_all() for sqlite then dispose on the end, that's all good
21:00:15 sean-k-mooney cdent: ah ok well im caling it a day anyway.
21:00:19 cdent zzzeek: when we talked about this before (on that review) you said that the existing fixtures wouldn't be quite adequate, which is how what's there ended up the way it did. you said it was "probably good enough|
21:01:00 zzzeek cdent: i belive that's because early on I thougth i was looking at nova :)
21:01:02 cdent zzzeek: but if there are ways it can be made more tidy, that would be great
21:01:18 zzzeek cdent: the oslo_db fixtures should work here fully and if they dont thats a bug in oslo.db i have to fix

Earlier   Later