Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-01
16:42:25 dansmith mriedem: I mean the destroying the BR and setting cell_id on the instance mapping in one transaction
16:42:39 dansmith melwitt: right
16:43:26 mnaser i assume mriedem is concerned about updating instance mapping and creating instance in a single transaction
16:43:43 mnaser cause you can create but fail updating instance mapping
16:43:51 mnaser and end up with a listable but not retrievable instance
16:43:59 mriedem that's what https://review.openstack.org/#/c/586713/ is for
16:44:16 mriedem we update the mapping immediately after creating the instance,
16:44:25 mriedem but i'm not sure what should happen if we fail to update the mapping
16:44:32 dansmith destroy the instance
16:44:37 dansmith leave the BR in place
16:44:44 mriedem that's what it's doing right now
16:44:45 mriedem my patch
16:44:51 dansmith that's what we should do
16:44:57 melwitt mnaser: yeah, would have to take a look at what we can do with finding that out. jaypipes do you know how we can print/log whether something is occurring in a single db transaction when we're doing sqlalchemy queries while we're testing something?
16:45:00 mriedem but,
16:45:00 dansmith if we double fail on all DBs then we can't do anything anyway
16:45:12 mriedem if we have an empty mapping and no instance and a build request,
16:45:23 dansmith melwitt: one writer should mean one transaction AFAIK
16:45:24 mriedem the build request will always show as status BUILDING or whatever
16:45:53 mnaser thing is stale records in db that don't impact api aren't too much of a problem
16:46:19 mnaser having an instance mapping point to a cell where the instance doesnt exist => no confusing instance when listing instances
16:46:20 dansmith mriedem: the other thing we can do, which I'd rather not, but is look up mappings during a list and either filter those instances or fix the mappings on the fly
16:46:24 mriedem i don't think you can delete a scheduling instance though, unless you force delete it?
16:46:46 mriedem dansmith: i thought about fixing on the fly,
16:46:46 dansmith mriedem: I mean NFO the record you just created
16:46:47 mriedem however,
16:47:09 mriedem if we create the instance but fail to update the mapping, we won't have created the bdms and tags in the cell db either
16:47:11 melwitt dansmith: it should, but IIRC there are ways in which queries might not be a single transaction (from a db standpoint) even if you put them under the same decorator. it would be nice to be able to verify if we could
16:47:16 mriedem so the instance could be mapped, but it would be incomplete
16:47:42 dansmith melwitt: only if you're doing nested transactions or force commit, AFAIK
16:48:07 jaypipes melwitt: you mean can we inject log messages after running various DB statements in the placement service to log whether there were results?
16:48:16 mriedem i don't know what NFO means, urbandictionary suggests Near Fatal Orgasm
16:48:18 jaypipes melwitt: not quite sure what you're asking, sorry...
16:48:19 dansmith mriedem: sure, but that's going to affect the listable instance anyway
16:48:24 dansmith mriedem: nuke from orbit
16:48:40 melwitt jaypipes: I mean is there a way to log/print/see whether a series of queries we made really translated into a single database transaction?
16:48:43 dansmith mriedem: although I like yours better
16:48:51 mriedem probably gonna need to serialize these conversations for context
16:48:57 melwitt jaypipes: in general, this isn't about placement
16:49:02 mnaser as an operator i would rather have an instance_mapping that's invalid pointing to an non-existant instance (but not showing any instances in list) rather than instance showing in list and not retrievable
16:49:05 jaypipes melwitt: ah, you mean a single COMMIT was issued...
16:49:37 mriedem dansmith: maybe you want to just look at https://review.openstack.org/#/c/586713/ and leave whatever thoughts
16:49:46 jaypipes melwitt: that would prolly be something we could add to oslo.db's transaction context manager. Basically, output a LOG.info("Committing transaction") when the context manager completes without an exception?
16:49:50 mriedem oh you did
16:50:12 mnaser my single transaction change was updated here -- https://review.openstack.org/#/c/586742/ .. but no testing etc
16:50:24 gnuoy melwitt, sorry, just so I'm clear, there needs to be an [api_database] section in the neutron.conf that neutron-metadata-agent is using?
16:50:32 dansmith mnaser: I know, but there are several things we're trying to accomplish with the dance around creating/destroying those records
16:50:47 dansmith mnaser: mostly around trying to preserve api semantics that have existed for a long time
16:51:19 mnaser dansmith: makes sense, i dont have that much context about it then :p
16:51:35 dansmith mnaser: mriedem: maybe we should have a non-text conversation about that whole process to make sure we're still doing what we want
16:51:54 dansmith I'm not likely to be able to focus on it if we do it here, I just have too many parallel threads running
16:52:23 melwitt jaypipes: I guess so :) just curious if there's a way to sanity check that in a unit test or func test environment (temporarily), the sql commands that are issued to the database. I had thought that it's not guaranteed that everything you put under one transaction context manager is a single transaction, from a low-level database standpoint. maybe there's no way to check that? anyway, this is all just my curiosity, so sorry for the
16:52:23 melwitt distraction
16:53:19 mnaser dansmith: i agree :)
16:55:08 mriedem yeah, also,
16:55:29 melwitt gnuoy: I'm not 100% sure but I think no, the neutron-metadata-agent is a neutron thing and the nova part is a metadata proxy that sends to the agent
16:55:40 mriedem i'd kind of like to squash these 2 for loops in schedule_and_build_instance for sanity before making changes in here, but not sure how backportable that all is, since we likely need at least one fix going back to pike
16:56:06 mriedem mnaser: are you still on pike?
16:56:07 dansmith melwitt: FYI I'm not going to be around for the nova meeting tomorrow. I'm sure I will be sorely missed.
16:56:31 mnaser mriedem: no, queens everywhere at this point
16:56:34 melwitt dansmith: yes, we will prepare ourselves emotionally
16:56:47 gnuoy melwitt, yeah I assumed querying the nova api db would be a nova task which is why the guide seems odd when it says that neutron needs it.
16:57:25 melwitt gnuoy: well, the guide is saying the neutron metadata API proxy (emphasis on the proxy) needs it. the nova piece that proxies to the neutron agent needs it
16:57:44 dansmith gnuoy: melwitt: there is no [api_database] in the _neutron_ config
16:58:03 jaypipes melwitt: no need to apologize! it's an interesting problem.
16:58:59 gnuoy is the guide trying to tell me that nova-api-metadata service needs an [api_database] section?
17:01:00 melwitt dansmith: do you know if the metadata service and the neutron metadata API proxy the same service? their code is together in the tree
17:01:27 dansmith melwitt: they neutron proxy is in the neutron tree no?
17:01:28 melwitt *are the same
17:02:13 dansmith melwitt: but gnuoy was asking about an api_database section in a neutron.conf file, which wouldn't be the case, unless neutron has a section for that as well
17:02:28 mnaser neutron does not have `api_database` section
17:02:32 dansmith right
17:02:34 mriedem neutron has a metadata agent that makes requests to the nova metadata api
17:02:41 dansmith right
17:02:57 mriedem https://docs.openstack.org/nova/latest/user/cellsv2-layout.html#neutron-metadata-api-proxy
17:02:58 dansmith I assume the confusion is over the configuration of the api_db for the _nova_ metadata side
17:03:05 mriedem "The Neutron metadata API proxy should be global across all cells, and thus be configured as an API-level service with access to the [api_database]/connection information."
17:03:09 dansmith separate from neutron's proxy of/to that
17:03:40 melwitt so what is the "neutron metadata API proxy" then? it's a nova service isn't it? is it our metadata service?
17:03:43 mriedem maybe that should say 'nova metadata api' to avoid confusion it's something in neutron
17:03:51 dansmith no
17:03:56 dansmith mriedem: yeah
17:04:18 mnaser melwitt: neutron has a 'software' that runs in the netns for each tenant/etc which listens of 169.254.169.254 and proxies those requests out to nova metadata api
17:04:35 dansmith right, which is the neutron metadata proxy
17:04:36 mnaser i assume thats the neutron metadata api proxy as i understand it
17:04:37 mriedem https://docs.openstack.org/neutron/latest/install/controller-install-ubuntu.html?highlight=metadata%20proxy#configure-the-metadata-agent
17:04:38 melwitt mnaser: thanks
17:04:52 mnaser but yeah, i think that doc should say Nova Metadata API rather than neutron metadata api proxy
17:05:03 gnuoy I will raise a PR for the docs
17:05:11 gnuoy Thanks everyone for the help
17:05:13 melwitt gnuoy: thanks. sorry for all that confusion
17:05:20 gnuoy np at all
17:05:25 dansmith mnaser: yeah, and apparently I wrote that
17:05:42 dansmith just a mind-o I'm guessing
17:05:43 mnaser writing long docs can get exhuasting :P
17:06:04 mnaser i'll have to deal with the image backend stuff again soon :(
17:06:19 melwitt godspeed
17:06:23 mriedem https://ibm-blue-box-help.github.io/help-documentation/nova/Metadata_service_FAQ/ is actually pretty nice
17:06:33 openstackgerrit Vishakha Agarwal proposed openstack/nova master: 'Updated_at' is NULL when show aggregate info https://review.openstack.org/580271
17:06:38 mnaser we're setting up a new az as a cell and i dont want glance downloading stuff 24x7 because we have an expensive 10g link across az's

Earlier   Later